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BACKGROUND 

ACES (Analysis Criteria Evaluation System) is a knowledge based expert system that automates 
the final certification of the Space Shuttle onboard flight software. Guidance, Navigation and 
Control of the Space Shuttle through all its flight phases are accomplished by a complex onboard 
flight software system. This software is reconfigured for each flight to allow thousands ot 
mission-specific parameters to be introduced and must therefore be thoroughly certified prior to 
each flight. This certification is performed in ground simulations by executing the software in the 
flight computers. Flight trajectories from liftoff to landing, including abort scenarios, are simu- 
lated and the results are stored for analysis. The current methodology of performing this analysis 
is repetitive and requires many man-hours. The ultimate goals of ACES are to capture the know- 
ledge of the current experts and improve the quality and reduce the manpower required to cerfity 
the Space Shuttle onboard flight software. 


FLIGHT SOFTWARE TEST DATA ANALYSIS 

The volume of data required to certify the flight software for each mission is enormous. Between 
15 and 25 simulations are run for each mission. Each simulation produces over 2 million plotted 
data points and over 30,000 lines of printed output. Analysis consists of manual evaluation ot: 

1. Sequences of flight activities 

2. Cockpit CRT displays 

3. Plotted flight software and simulator parameters 

ACES automates the analysis of this data by using the same methodology that the experts use. 
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KNOWLEDGE ENGINEERING AND IMPLEMENTATION 

The first phase of knowledge acquisition began by documenting the criteria that the experts use to 
analyze each of the testcases. This knowledge was written as pass/fail criteria for the sequences, 
displays and parameters in each test case. They were written such that they could be used manu- 
ally by inexperienced analysts. Concurrently, automation techniques were evaluated and proto- 
typed. 

The pass/fail criteria were first prototyped in a mainframe rule-based expert system shell, Expert 
System Environment (ESE). ESE was not used because of the volume of data to be processed. 
Instead, three PL/I pre-processors were created to evaluate the pass/fail criteria for the three dif- 
ferent types of data to be analyzed. 

1. Flight Equipment Interface Device Online Evaluator (FOE) analyzes the sequences of events 
throughout the trajectory 

2. Display Electronics Unit Criteria Evaluator (DEUCE) analyzes the cockpit displays 

3. Automation Program (Auto Prog) analyzes the plotted parameters 

The outputs of the pre-processors are the data items which violated the criteria limits as well as 
additional data needed for evaluation of the violations. These pre-processors exhibit expert 
system qualities in that they perform the preliminary analysis that the experts perform. 

The initial ACES prototype was enhanced with additional rules to analyze the criteria violations. 
The methodology for evaluating the pass/fail criteria violations was implemented in another 
mainframe rule-based expert system shell, KnowledgeTool (KT). KT was chosen because of its 
speed, modularity and ability to interface with other programs. In ACES, KT runs in batch 
mode on IBM 30xx MVS processor mainframes. The process of running the pre-processors and 
expert system for data analysis is automated and no manual intervention is required except to 
submit the job. 

The following diagram illustrates the data flow for ACES. The process begins with the simulation 
execution. The data is stored on a logtape and then analyzed by the three pre-processors. The 
results are criteria violations and additional data needed by the KT knowledge base which are 
sorted and sent to KT. The KT source code is compiled into PL/I executable modules and rule 
files which are then linked to the KT executable code to run the application. The final results are 
viewed on the analyst's terminal. After the analyst determines the cause of the problems, the 
knowledge base can be updated so that it can Team* the new information which would then be 
used for future shuttle flights. 
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SYSTEM VALIDATION/VERIFICATION 

Validation of ACES was separated into two phases: 

• Pre-processor Tools Verification 

• Knowledge Validation 

The pre-processor tools were verified using the standard requirements, design and code review 
techniques. All of the capabilities of each tool were exercised and the results were reviewed by a 
team including the experts. 

The knowledge base rules were validated dynamically by having the experts compare the results 
produced by ACES to those of manually certified simulations. All differences were resolved and 
changes were made to the pre-processor tools, the pre-processor rules or to the Knowledge tool 
rules Once the validation was complete for the rules were baselined and put under configuration 

control. 
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BENEFITS OF ACES 


• Captured expert knowledge 

• Reduced total time to certify the flight software 

• Increased consistency in data interpretation 

• Improved quality of the data analysis 

• Reduced dedicated and elapsed time to analyze each simulation 


CURRENT STATUS AND FUTURE DEVELOPMENT 


A CE S is being used in production mode by the Flight Software Certification organization to 
certify the Shuttle Onboard FUght Software for the Deorbit trajectory. Some of the criteria are 
still analyzed manually while they are being coded into ACES. For the remaining cases, inputs to 
Auto rrog are being baselined and the knowledge bases are being coded into KT. Research is 
being conducted to determine how the inference engine and the frames referencing capabilities of 
KT can be used most effectively. 


ACES shows that expert systems can be successfully used to certify/verify complex software 
systems. Knowledge engineering and acquisition requires considerable time and effort but the 
retention of the expertise has proved to be very valuable in many areas including analysis consist- 
ency and training. Overall, time and resources can be reduced while the quality of analysis is 

maintained or improved by inserting expert system technology into existing software testing envi- 
ronments. 
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